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A  SURVEY  OF  REMOTE  MONITORING 


Abstract 


This  report  describes  remote  monitoring  in  the  application  areas  of 
performance  evaluation,  diagnostic  testing,  performance  assurance 
and  system  security  testing.    The  evolution  of  remote  monitoring  is 
briefly  reviewed  and,  then,  remote  monitors  are  categorized  into 
seven  classes.    Several  example  systems  are  discussed  for  each 
classification,  along  with  their  capabilities  in  each  application 
area.    The  views  presented  in  this  report  represent  only  those  of 
the  author,  an  independent  consultant,  and  should  not  be  construed 
as  a  policy  statement  of  NBS  or  any  other  organization. 
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remote  monitoring;  system  security  testing 
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INTRODUCTION 


The  general  area  of  remote  monitoring  of  computer  systems  encompasses  a 
broad  spectrum  of  mechanisms  for  a  wide  variety  of  purposes.    In  this 
report,  the  discussion  is  restricted  to  monitoring  systems  or  studies 
where  a  mechanism  is  used  to  measure  or  observe  the  performance  of  a 
computer  system,  and  that  mechanism  can  be  controlled  by  another  device 
or  a  human  from  some  geographically  distinct  location.    In  most  cases, 
it  is  expected  that  the  monitoring  device  itself  is  designed  to  collect 
data  about  the  host  system,  perform  at  least  preliminary  filtering  of 
the  raw  data,  and  then  either  store  the  filtered  data  for  retrieval  by 
the  central  controller  or  immediately  transmit  the  filtered  data  to  the 
central  monitor  controller.    The  nomenclature  used  for  the  various  con- 
stituents, then, is  as  follows:    The  host  system  is  the  installation 
being  monitored;  a  monitor  that  is  local  to  the  host  is  referred    to  as 
a  remote  monitor.      The  remote  monitor  is  ultimately  controlled  from  a 
central  site  by  the  central  monitor  controller.    The  host  computer  is 
considered  to  be  the  remote  facility,  while  the  measurement  control  and 
analysis  take   place  at  the  central  site. 

This  classification  of  remote  monitors  admits  such  approaches  as:  those 
implemented  purely  in  software  which  can  be  interrogated  from  an  external 
terminal,  programmable  hardware  monitors,  hardware  monitors  distributed 
over  different  portions  of  the  host  machine,  hybrid  monitors,  monitors 
used  in  distributed  computer  networks,  fault  diagnosis  monitors,  and 
extended  consoles  for  a  computer  system.    Each  of  these  categories  will 
be  discussed  in  detail  in  a  later  section  of  this  report.    The  classifi- 
cation excludes  classic  hardware  monitors  that  require  plugboard  alter- 
ations to  change  the  logical  combination  of  probe  signals.    It  also 
excludes  pure  software-implemented  monitors  which  use  the  normal  oper- 
ating system  facilities  for  "triggering,"  reporting  and  recording. 

Remote  monitors  are  being  used  in  a  number  of  ways  that  earlier,  locally 
controlled  monitors  were  not  used.    The  most  obvious  use  of  the  monitors 
is  for  gathering  performance  data.    Remote  performance  monitor  facil- 
ities are  frequently  divided  into  a  number  of  remote  data  gathering 
mechanisms  plus  a  single,  shared  facility  to  analyze  data  ggd  prepare 
reports;  the  Tesdata  facilities  are  examples  of  this  type.  Distri- 
buted monitors,  perhaps  best  exemplified  by  the  PARTNER  package  for 
Control  Data  6000  series  machines,      are  also  frequently  used  for  per- 
formance measurements;  the  idea  here  is  to  dedicate  certain  hardware  fac- 
ilities of  the  host  system  to  the  measurement  function.  Programmable 
hardware  monitors  are  merely  a  refinement  of  earlier  hardware  monitors, 
and  also  are  primarily  used  for  performance  measurement. 

A  newer  application  of  remote  monitors  is  for  computer  system  diagnosis 
and  remote  exercising  of  a  computer  system,    A  number  of  computer  manu- 
facturers have  included  this  capability  in  their  current  product  line. 
5,32,47,58,59    j^q  basic    idea  is  to  replace  the  conventional  operator's 
console  with  an  intelligent  device  such  as  a  minicomputer.    The  intelli- 
gent  console  can  be  used  to  inspect  any  of  a  number       conditions  that 

Superscript  numbers  indicate  literature  references  at  end  of  the  report. 
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exist  in  the  host  machine,  allowing  the  observed  condition  to  be  analyzed, 
recorded  or  transmitted  on  a  telecommunications  link  to  a  remote  controller. 
The  remote  controller  may  be  a  human  operator  or  another  computer  system. 
Although  this  is  apparently  a  new  concept  to  most  machine  manufacturers, 
it  should  be  noted  that  Control  Data  6000  series  computer  systems  have^ 
used  this  approach  to  implement  their  consoles  for  a  number  of  years. 

A  new  area  for  which  remotely  controlled  monitors  might  be  employed  is 
that  of  performance  assurance  and  safeguard  studies.    The  goal  is  to  mon- 
itor the  workload  of  a  computer  system  in  order  to  either  assure  a  given 
level  of  performance,  or  to  assure  that  a  computer  system  is  not  being 
used  for  tasks  that  were  not  intended  to  be  executed  on  that  system. 

Although  it  would  be  satisfying  to  be  able  to  monitor  a  processor's  pro- 
gram counter  to  determine  what  program  the  processor  is  executing,  this 
is  obviously  impossible  in  the  general  case.    It  is  easy  to  construct  an 
example  that  shows  that  if  one  could  write  an  algorithm  that  inspects  the 
program  counter  locus  and  identifies  a  corresponding  algorithm,  then  one 
ought  to  be  able  to  write  a  similar  algorithm  that  inspects  the  program 
counter  locus  and  indicates  whether  the  corresponding  algorithm  will  ever 
terminate  or  ragt.    The  latter  algorithm  has  been  proven  to  be  impossible 
to  construct.       Nevertheless,  there  are  other  activities  in  the  computer 
system  that  can  be    observed  with  a  monitor,  e.g.  resouce  utilization. 
One  can  easily  compute    the  ratio  of  input/output  time  to  central  pro- 
cessor time  for  a  given  job.    This  will  allow  one  to  partition  heavy 
computer  jobs  from  input/output  bound  jobSo 

Although  it  is  impossible  to  identify  arbitrary  programs  in  execution,  it 
may  be  possible  to  recognize  a  small  set  of  programs  when  they  are  execu- 
ting on  the  host  system.    For  example,  suppose  that  an  installation  is 
intended  to  only  execute  programs  P, ,  P^...  P    (on  arbitrary  data).  It 
may  be  possible  to  employ  heuristic  technique?  to  recognize  exactly  when 
one  program  from  that  set  executes,  while  any  unrecognizable  program  is 
declared  to  be  illegal.    In  this  case,  remote  monitoring  techniques  can 
be  used  to  recognize  the  "signature"  of  each  of  the  n  acceptable  programs. 

Finally,  remote  monitors  may  be  used  to  enhance  system  security  or  to 
provide  a  mechanism  for  checking  the  security  of  a  system.    It  is  clear 
that  the  presence  of  monitors  of  any  type  are  a  threat  to  the  overall 
security  of  a  computer  system,  e.g.  see  references  6  and  13.    Whenever  a 
mechanism  (i.e.  a  monitor)  is  provided  the  capability  of  observing  criti- 
cal portions  of  the  operating  system,  then  that  same  device  can  be  mali- 
ciously employed  to  penetrate  the  conventional  security  mechanisms  of 
that  system.    By  partitioning  a  monitor  into  a  local  internal  component 
and  a  remote  external  component,  system  security  has  a  much  better  chance 
of  being  effective,    ".'he  internal  monitor  can  be  written  as  an  internal 
portion  of  the  operating  system  itself,  subject  to  the  same  design  con- 
straints (such  as  proof  of  correctness,  restricted  entry  points,  author- 
ized access,  etc.)  as  other  modules.    The  attendant  software  is  essen- 
tially data-gathering  code,  which  is  simpler  and  easier  to  make  secure 
than  a  full  software  monitor.    The  external  portion  of  the  monitor  is 
allowed  to  access  the  internal  portion  through  normal,  secure  paths,  thus 
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allowing  authorization  checks  and  entries  into  predefined  procedures  of 
the  operating  system. Although  this  approach  is  not  totally  secure, 
it  offers  a  much  more  effective  security  policy  than  undisciplined  mon- 
itoring of  the  host  system. 

Another  variant  of  remote  monitors  can  be  used  to  audit  a  computer  sys- 
tem's security  state.    The  basic  idea  is  to  distribute  a  monitor  of 
internal  and  external  components,  as  above.    The  external  portion  is  used 
interactively  by  a  human  that  is  responsible  for  system  security  to  audit 
various  portions  of  the  machine  with  the  aid  of  the  internal  portion  of 
the  monitor.    This  approach  is  used  in  the  WWMCCS  computer  systems, and 
will  be  discussed  at  length  in  the  body  of  this  report.    The  Rand  Corpor- 
ation has  also  investigated  the  use  of  monitors  to  detect  data  bank  intru- 
sions and  to  delay  the  intruder  until  other  protective  action  can  be 
taken.  ^-^ 

In  the  remainder  of  this  report  the  background  of  remote  monitoring  will 
first  be  examined.    The  evolution  of  present-day  monitoring  systems  will 
be  traced  from  early  performance  monitoring  work.    The  main  body  of  this 
report  is  the  next  section;  seven  categories  of  remote  monitors  are 
defined,  and  a  number  of  examples  of  each  category  are  discussed.  The 
final  section  draws  some  conclusions  about  capabilities  and  limitations 
for  various  application  areas  and  looks  briefly  at  future  trends  in  remote 
monitoring. 

BACKGROUND 

In  this  section  of  the  report,  the  evolution  of  remote  monitors  is  dis- 
cussed beginning  with  hardware  and  software  monitors  of  the  1960-1970 
era.    In  the  early  1970's  monitoring  techniques  and  tools  became  sub- 
stantially more  sophisticated,  leading  to  the  development  of  mechanisms 
that  could  be  construed  as  remote  monitors.    This  section  will  briefly 
describe  this  evolution  into  current  remote  monitoring  technology. 

Computer  system  monitoring  has  become  a  primary  component  of  system 
design,  manufacture,  and  maintenance  because  of  its  application  to 
performance  evaluation.    Although  testing  instruments  (e.g.  oscilloscopes) 
were  frequently  used  to  monitor  the  hardware  at  a  very  low  level,  system 
monitoring    did  not  really  begin  to  be  needed  until  the  mid  1960's.  In 
the  early  part  of  that  decade,  computer  systems  began  to  reach  a  level  of 
sophistication  where  resources  were  shared  among  a  set  of  users.  Once 
resource  sharing  was  introduced,  then  resource  utilization  became  an 
important  metric  for  that  system.    If  utilization  was  too  high,  then  the 
resource  represented  a  bottleneck  to  system  progress;  if  utilization  was 
too  low,  then  the  resource  was  either  over-configured  or,  perhaps,  was 
being  prevented  from  being  used  effectively  by  bottlenecks  elsewhere  in 
the  system.    The  result  was  frantic  activity  in  the  areas  of  hardware 
and  software  monitor  development. 
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Software  Monitors 


Software  monitors  have  not  changed  significantly  in  the  last  ten  to  fif- 
teen years,  although  the  designs  have  become  more  intricate.    The  pre- 
dominante  idea  behind  a  software  monitor  is  that  the  operating  system  of 
the  host  machine  is  modified  so  that  it  will  collect  and  save  measurement 
data  about  the  performance  of  the  machine.    There  are  three  major  tech- 
niques for  implementing  a  software  monitor  (discussed  at  length  else- 
where).'''^"   The  first  technique  is  to  use  the  system  log  as  a  storage 
medium  for  recording  the  occurrence  of  events.    The  system  log  may  sub- 
sequently be  analyzed  to  determine  the  activity  of  the  machine  at  a  level 
dictated  by  the  post  processor  and  the  raw  data  recorded  on  the  system 
log.    This  approach  has  proven  to  be  a  cost-effective  mechanism  for  per- 
formance monitoring  at  a  gross  level 

For  more  detailed  studies,  a  more  sophisticated  data-gathering  tool  must 
be  employed.    The  system  log  cannot  normally  be  used  to  trace  the  program 
counter  locus.    In  these  cases,  more  extreme  modifications  must  be  made 
to  the  host  operating  system  in  order  to  invoke  specially  written  mon- 
itoring and  recording  code.    One  technique  to  do  this  is  the  interrupt- 
intercept  method.    This  technique  assumes  that  the  only  times  at  which 
measurements  should  be  taken  are  when  the  system  changes  states.    In  an 
interrupt-driven  operating  system,  (e.g.  IBM's  OS)  state  changes  are  ini- 
tiated by  an  interrupt  (or  a  trap).    At  each  such  occurrence,  the  CPU  is 
restarted  on  a  handler,  usually  as  a  function  of  the  type  of  interrupt. 
The  interrupt-intercept  monitor  modifies  this  interrupt  address  so  that 
pertinent  interrupts  are  directed  to  monitoring  code  rather  than  to  the 
handler.    The  monitoring  code  records  the  event  and  then  branches  to  the 
handler.    The  raw  data  is  subsequently  recorded  on  a  mass  storage  device 
and  analyzed  offline.    (IBM  has  successfully  used  the  technique  in  OS 
measurements,^^  as  did  GE  in  the  GCOS  operating  system, as  well  as 
others. ) 

While  interrupt-intercept  monitors  can  be  used  successfully  to  monitor 
events  correlated  with  an  interrupt  or  a  trap,  they  cannot  be  success- 
fully used  in  a  machine  which  does  not  incorporate  an  interrupt-driven 
operating  system  (e.g.  Control  Data  Cyber  machines) .    An  alternative  is 
to  use  random  time  intervals  for  invoking  the  measurement  routines;  this 
technique  has  been  used  in  many  different  cases,  e.g.  see  references  24 
and  54. 

Software  monitors  have  several  assets,  including  their  ability  to  be 
implemented  without  modifying  existing  hardware.    Unfortunately,  they  do 
tend  to  add  time  and  space  artifact  to  the  host  system,  producing  a  halo 
effect  to  the  measurement  experiment.    The  resolution  of  a  software  mon- 
itor is  always  limited  by  the  instruction  repertoire  and  cycle  time  of 
the  host  machine.    A  software  monitor  may  also  be  corrupted  or  bypassed 
by  processes  that  do  not  wish  to  be  monitored. 

Hardware  Monitors 

The  development  of  hardware  monitors  lagged  that  of  software  monitors  by 
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a  few  years,    at  least  in  the  public  domain.    Drummond  discusses  several, 
hardware  monitors  that  were  used  internally  at  IBM  on  the  7094  systems,'' 
while  customers  were  still  making  crude  attempts  at  monitoring  in  the 
late  1960's.48    ind  ependent  vendors  began  constructing  special -purpose 
hardware  monitors  in  1967-1968,  propagating  a  large  number  of  such  firms. 
Currently,  only  two  such  firms  still  exist  (i.eo  have  any  significant 
portion  of  the  market) --Comten  and  Tesdata. 

Early  hardware  monitors  consisted  of  three  major  components:  a  signal 
filter  and  combination  unit,  a  time  and  count  unit,  and  a  data  recording 
unit.    The  signal  filter  and  combination  unit  was  usually  made  up  of  a 
set  of  high  impedance  probes  which  could  be  attached  to  the  logic  boards 
of  the  host  machine  without  disturbing  the  existing  electronic  signals 
on  those  logic  boards.    The  signals  sensed  by  the  probes  were  passed  to 
a  logical  combination  unit,  which  could  be  used  to  detect  the  simultaneous 
occurrence  of  two  or  more  probe  signals,  the  exclusive  occurrence  of  one 
signal  from  among  a  set  of  signals,  etc.    The  particular  logical  combina- 
tions chosen  were  "programmed  into"  the  hardware  monitor  by  conventional 
plugboard  logic.    Once  an  event,  or  a  logical  combination  of  events,  had 
been  detected,  the  occurrence  or  duration  of  that  event  was  temporarily 
recorded  in  the  time  and  count    unit.    This  unit  was  ordinarily  implemen- 
ted as  a  set  of  addressable  registers,  although  more  sophisticated  mon- 
itors began  to  incorporate  content  addressable  memories.-^' Inasmuch 
as  these  registers  were  accumulating  data,  they  were  subject  to  overflow; 
therefore,  their  contents  were  frequently  stored  on  the  data  recording 
unit  (usually  a  tape  drive). 

Hardware  monitors  of  the  generation  described  above  had  the  ability  to 
take  measurements  of  a  computer  system  at  a  much  finer  resolution  than 
any  software  monitor;  furthermore,  they  created  no  time  and  space  arti- 
fact.   The  monitors  were  also  not  reachable  from  any  software  in  the 
host  machine;  hence,  they  were  protected  from  corruption  and/or  bypassing. 
However,  it  was  difficult  to  draw  a  correspondence  between  observed  mon- 
itor data  and  a  particular  task  or  procedure  executing  on  the  host  system, 
i.e.  causal  relationships  were  lost.    The  amount  of  raw  data  collected 
also  tended  to  be  too  large  for  reasonable  offline  analysis  (and  online 
storage).    Thusj  the  strengths  of  pure  hardware  monitors  were  the  weak- 
nesses of  software  monitors,  and  vice  versa. 

Intelligent  Monitors 

The  first  significant  step  in  the  direction  of  improving  the  chasm  be- 
tween software  and  hardware  monitoring  techniques  turns  in  the  general 
direction  of  remote  monitoring  (as  discussed  in  this  report).    The  moni- 
tor might  be  distributed  across  special -purpose  measurement  hardware  and 
software  internal  to  the  host  operating  system.    In  1967,  Estrin  et  al. 
published  a  paper  describing  the  SNUPER  COMPUTER  monitoring  system  pro- 
posed for  facilities  at  UCLA. 12    The  SNUPER  COMPUTER  system  was  a  complete 
system  based  around  a  SDS  (XDS)  Sigma  7  processor  with  16K  of  32-bit 
memory.    The  input  to  the  processor  was  from  a  "sensory  system"  via  nor- 
mal data  channels  and  high-speed  I/O  pathSo    The  sensory  system  design 
would  ultimately  include  a  filtering  processor  to  analyze  raw  monitor 
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data,  generate  event  counts  and  measure  the  duration  of  events.  Details 
of  the  design  for  SNUPER  COMPUTER  components  were  never  published;  and, 
apparently,  the  entire  instrumentation  system  was  never  operational. 
Nevertheless,  this  paper  was  the  first  published  report  of  a  sophisticated 
monitoring  system  in  which  the  instrumentation  facility  was  centered 
around  a  programmable  processing  unit,  allowing  the  hardware  monitor  to 
dynamically  control  a  measurement  session.    Drummond  indicates  that  IBM 
was  using  one  computer  to  measure  anothec.,in  the  early  1960's  with  the 
Direct  Couple  (7040-7090)  system.'''  ^P  '^^'^ 

The  general  state-of-the-art  in  the  late  1960's  began  to  see  intelligent 
hardware  monitors  which  employed  a  simplified  version  of  the  SNUPER  COM- 
PUTER approach.    These  monitors  were  still  pure  hardware  monitors  in 
which  the  static  measurement  experiment  was  determined  a  priori.  However, 
a  processor  (usually  a  minicomputer)  was  employed  to  logically  combine 
event  signals,  to  manage  event  counters  and  timers  in  software,  to  pro- 
vide a  programmable  filter  for  raw  performance  data,  and  to  control  the 
data  recording  function.    Additionally,  such  hardware  monitors  typically 
provided  online  displays  to  describe  the  state  of  the  host  machine  in  real 
time.    (Notice  that  this  point  of  evolution  essentially  corresponds  to 
the  use  of  a  remote  monitor  as  a  system  console  as  mentioned  in  the  Intro- 
duction.)   During  this  period  of  development,  most  of  the  hardware  compo- 
nents had  been  incorporated  into  monitors  so  that  they  could  interact  with 
host  system  software;  but,  for  a  few  years,  the  monitors  were  not  used  in 
that  manner.    Once  a  programmable  monitor  is  used  to  query  the  status  of 
the  host  in  order  to  dynamically  control  the  monitoring  function,  that 
monitoring  system  has,  in  effect,  become  a  remote  monitoring  system.  The 
control  element  corresponds  to  the  minicomputer-based  monitor,  and  the 
remote  portion  of  the  monitor  is  that  part  of  the  host  system  which  passes 
information  to  the  external  monitor.    These  applications  of  minicomputer- 
based  hardware  monitors  are  discussed  in  detail  in  the  next  section  of 
this  report. 


REMOTE  MONITORING  TECHNIQUES 

In  the  late  1960's  and  early  1970's,  techniques  for  remote  monitoring  of 
computer  systems  were  beginning  to  be  explored,  many  of  them  based  on 
existing  hardware  and  software  technology.    The  state-of-the-art  at  that 
time  was  discussed  in  the  previous  section  of  this  reporto    In  the  current 
state  of  development,  there  are  approximately  seven  classifications  of 
remote  monitors: 

-  Remotely  controlled  software  monitors 

-  Internally  distributed  monitors 

-  Programmable  monitors 

-  Hybrid  monitors 

-  Computer  network  monitors 

-  Fault  diagnosis  monitors 

-  Intelligent  and  extended  consoles 
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Remotely  controlled  software  monitors  are  pure  software  monitors  in  terms 
of  their  techniques  for  obtaining  measurement  data;  they  differ  from  con- 
ventional software  monitors  in  their  ability  to  be  controlled  from  an 
external  source  (e.g.  an  interactive  terminal).    Internally  distributed 
monitors  use  a  combination  of  software  and  hardware  resources  to  take 
measurements,  where  all  hardware  resources  are  parts  of  a  conventional 
(uninstrumented)  computer  system.    That  is,  certain  built-in  hardware 
facilities  are  used  for  monitoring.    Programmable  hardware  monitors  are 
extensions  of  those  discussed  in  the  background  section,  and  hybrid  moni- 
tors are  the  logical  conclusion  of  the  programmable  hardware  monitors. 
The  monitor  components  are  distributed  across  remote  internal  (software) 
and  remote  external  (programmable  hardware)  portions  under  the  control  of 
a  local  facility.    Computer  network  monitors  are  incorporated  into  a  con- 
ventional network  (such  as  the  ARPANET)  in  order  to  monitor  communication 
and  node  performance.    Fault  diagnosis  monitors  are  characterized  as  any 
devices  used  to  check  hardware  machine  state  for  circuit  consistency,  etc.; 
a  degenerate  example  of  a  fault  diagnosis  monitor  is  a  logic  analyzer  or 
oscilloscope.    Intelligent  and  extended  consoles  are  constructed  from 
programmable  devices  which  serve  as  a  conventional  operator's  console 
under  "normal"  operation  and  as  a  pseudo  hybrid  monitor  during  monitoring 
sessions . 

This  characterization  of  remote  monitors  admits  to  imprecision  in  the 
sense  that  many  remote  monitors  could  be  classified  into  two  or  more  of 
the  named  divisions;  and,  in  fact,  several  particular  studies  will  be 
mentioned  under  more  than  one  category.    The  remainder  of  this  section  is 
made  up  of  more  detailed  discussions  of  the  above-mentioned  categories. 


Remotely  Controlled  Software  Monitors 

A  remotely  controlled  software  monitor  can  be  implemented  as  simply  as 
a  system  log  monitor.    The  purpose  of  the  monitor  is  limited  only  by  the 
ingenuity  of  the  impl ementers ,  e„g.  it  may  monitor  the  CPU  utilization, 
inspect  resource  utilization,  etc.    In  a  pure  software  monitor,  the  moni- 
tor is  instigated,  for  a  particular  run,  by  an  operator,  a  system  clock, 
or  it  is  set  for  cyclic  invocation  at  system  initiation  time.    A  remotely 
controlled  software  monitor  is  triggered  by  a  central  monitor  controller. 
There  are  two  prominent  examples  of  actual  implementations  which  employ 
this  technique:  the  first  example  is  the  WWMCCS  ADP  System  Security 
Officer  (WASSO)  station, 29j42  gp^  ^1^^  second  example  is  a  technique  used 
on  Control  Data  computers. ^ ^' 

The  WASSO  station  potentially  could  be  implemented  as  a  true  remotely 
controlled  software  monitor,  or  as  a  monitor  distributed  between  a  soft- 
ware tool  and  an  intelligent  terminal;  the  former  case  is  discussed  in 
this  section. 

Most  sites  in  the  WWMCCS  are  centered  around  a  Honeywell  6000  series 
computer  under  the  GCOS  III  operating  system.    Each  site  is  potentially 
processing  sensitive  data  in  a  multiprogramming  environment;  thus,  there 
is  a  need  for  a  secure  operating  policy  in  the  system.    Each  such  site 
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has  at  least  one  ADP  System  Security  Officer  whose  mission  is  to  monitor 
the  physical  protocol  of  the  site,  as  well  as  the  internal  operation  of 
the  system.    The  WASSO  station  is  under  development  to  aid  the  officer 
in  the  latter  part  of  his  mission  by  providing  a  facility  to  implement 
collection,  reduction,  and  analysis  of  information  necessary  to  assess 
the  site's  security  posture. 

The  current  status  of  the  WASSO  study  is  that  a  prototype  system  is  being 
built,  and  this  prototype  will  partially  rely  on  the  standard  software 
tools  that  exist  in  the  operating  system.    The  GCOS  III  operating  system 
has  been  declared  to  be  inadequate  to  support  the  desired  security  pol- 
icies necessary  for  WWMCCS;  thus,  a  new  operating  system  will  ultimately 
be  designed  and,  presumably,  that  operating  system  will  incorporate  new 
software  monitoring  tools.    The  current  software  tools  include  facilities 
to  designate  one  time  sharing  terminal  as  a  "master  terminal"  to  be  used 
by  the  WASSO.    From  this  master  terminal,  one  can  monitor  system  status 
information,  line  and  terminal  control  tables,  as  well  as  inspect  all 
interactions  between  the  computer  and  an  interactive  user.    The  WASSO  can 
also  inspect  memory  utilization,  mass  storage  utilization  and  file  organi- 
zation via  the  batch  stream  (i.e.  offline  analysis  similar  to  system  log 
analysis).    A  number  of  other  standard  measurement  tools  are  available  to 
the  WASSO,  all  of  which  are  implemented  as  conventional  software  monitors. 
The  proposed  WASSO  station  distributes  some  of  the  monitoring  function 
onto  a  tailor-made  intelligent  terminal.    This  extension  to  the  facilities 
of  the  WASSO  clearly  puts  the  mechanism  into  the  class  of  hybrid  monitors; 
however,  it  will  be  discussed  in  this  section  for  the  sake  of  continuity 
(and  then  cross-referenced  in  later  sections). 

The  WASSO  station  incorporates  a  set  of  monitor  data  I/O  buffers,  a  pro- 
cessor to  analyze  data  and  a  full  set  of  peripherals  used  to  report  and 
record  monitor  data.    The  station  is  expected  to  be  implemented  on  a 
Honeywell  Level  6  minicomputer  system,  including  48K  words  of  memory, 
diskettes,  magnetic  tape,  card  reader,  line  printer,  a  cartridge  disk  unit, 
and  the  console.    Functionally,  the  station  will  maintain  system  access 
controls,  alter  job  priorities,  explicitly  interconnect  the  Honeywell 
6000  node  into  the  WWMCCS  network  and  handle  detected  security  breaches. 
The  critical  observation  here  is  that,  again,  host  software  is  used  to 
actually  collect  the  data  used  for  determining  the  activity  of  the  host 
computer  system. 

Control  Data  employs  pure  software  techniques  in  a  manner  that  is  unique 
due  to  the  architecture  of  their  Cyber  series  systems.    A  detailed  des- 
cription of  the  technique  perhaps  best  belongs  in  the  section  on  inter- 
nally distributed  monitors,  since  it  employs  a  peripheral  processor  to 
monitor  the  state  of  the  remainder  of  the  machine.    The  aspect  of  the 
work  that  makes  it  appropriate  for  this  section  is  that  software  monitor- 
ing of  a  host  system  can  be  initiated  and  controlled  from  a  central  site, 
while  the  data  collection  and  analysis  are  performed  at  the  remote  site 
by  (a  portion  of)  the  host  system  itself.    See  the  next  section  for  the 
technical  description. 
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Another  example  of  remotely  controlled  software  monitoring  was  implemented 
at  MIT  in  the  late  1960's. The  monitor  probes,  and  parts  of  the  filter- 
ing and  analysis,  were  implemented  in  the  host  system  software,  while 
online  displays,  further  analysis,  and  data  recording  were  implemented  on 
a  central  site  minicomputer,    A  discussion  of  this  study  introduces  the 
subsection  on  hybrid  monitors. 

Remotely  controlled  software  monitoring  can  be  a  cost-effective  method 
for  implementing  a  remote  monitoring  facility  in  certain  situations.  If 
the  environment  of  the  host  system  is  "friendly,"  then  the  likelihood  of 
a  successful  implementation  for  performance  evaluation  or  diagnostic 
testing  is  good.    However,  if  the  goal  of  monitoring  is  performance  assur- 
ance or  system  security,  pure  software  techniques  may  be  unsuccessful  due 
to  the  well-known  complexity  of  software  in  the  present  day  and  age.  Even 
in  an  apparantly  friendly  environment  (i„e.  there  is  no  need  for  performance 
assurance  nor  security  enforcement),  it  is  almost  certain  that  the  software 
monitor  can  be  bypassed  and/or  violated.    (Although  there  is  no  proof  that 
the  above  statement  is  true,  this  author  knows  of  no  completely  secure 
software  system.)    In  a  potentially  unfriendly  environment,  the  probability 
of  successfully  implementing  a  completely  safe  remotely  controlled  soft- 
ware monitor  is  vanishing,  no  matter  what  the  reason  for  monitoring. 


Internally  Distributed  Monitors 

An  internally  distributed  monitor  employs  portions  of  the  host's  hardware 
in  conjunction  with  (possibly)  special -purpose  software  to  monitor  the 
operation  of  the  host  itself.    If  means  are  provided  by  which  a  user  can 
remotely  stimulate  the  internally  distributed  monitor,  then  it  can  be 
characterized  as  a  remote  monitoro    (The  distinction  between  remotely 
controlled  software  monitors  and  internally  distributed  monitors  is  not 
well-defined;  similarly,  hybrid  monitors  and  some  internally  distributed 
monitors  bear  strong  similarities.)    Ordinarily,  only  machines  with  mul- 
tiple   processors  can  provide  a  hardware  environment  for  implementing 
internally  distributed  monitors;  however,  note  that  the  multiple  processors 
need  not  be  homogeneous.    A  number  of  different  domestic  companies  incor- 
porate multiple  processors  into  their  product  line  (e.g.  Digital  Equipment 
PDP  10  series  computers.  Control  Data  Cyber  series  computers,  Univac  1100 
series  computers.)    There  have  also  been  some  machines  modified  by  indi- 
vidual research  organizations  so  that  they  can  support  internal  distri- 
buted monitoring,  e.g.  see  references  20  and  34„ 

The  approach  used  for  internally  distributing  a  monitor  is  a  simple  one: 
One  processor  of  the  host  system  is  temporarily  reserved  for  use  as  a 
hardware  monitor  processor,  while  the  remaining  processors  are  employed 
by  the  host  in  a  conventional  manner^    The  detached  processor  simulates 
the  action  of  the  signal  filter  and  combination  unit,  the  time  and  count 
unit,  and  the  analysis  and  recording  unit  of  a  conventional  hardware  moni- 
tor.   The  primary  failing  of  the  simulation  is  that  hardware  probes  are 
ordinarily  not  used  to  gather  data  for  the  simulated  hardware  monitor. 
Instead,  software  executes  in  other  parts  of  the  host,  as  well  as  in  the 
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detached  processor,  to  perform  the  data  gathering  function.    The  approach 
can  be  understood  in  its  entirety  by  considering  an  example  employed  on 
the  Control  Data  6000  architecture  by  indi vidual s as  well  as  by  the 
vendor .15,25,41,65 

The  Control  Data  6000  series  machines  include  one  or  two  central  process- 
ing units  (CP's)  and  ten  peripheral  processing  units  (PR's).    A  CP  is 
used  for  60-bit  general  purpose  computation  while  each  PP  can  be  used  as 
an  independent  12-bit  processor  to  perform  I/O  operations,  implement  the 
heart  of  the  executive  and  control  the  system  displayo    CP-PP  communica- 
tion takes  place  through  the  central  memory  of  the  machine;  the  machine 
state  is  also  maintained  in  the  central  memory. 

The  orthodox  mode  of  operation  for  the  overall  machine  intends  for  user 
(application)  programs  to  execute  on  a  CP,  where  general  purpose  computa- 
tions can  be  implemented  under  high  level  language  program  control.  When- 
ever the  operating  system  needs  to  intervene  with  the  user  processing,  or 
whenever  the  appl ication  program  needs  to  perform  an  I/O  operation,  then 
that  activity  is  executed  on  a  PP.    Thus,  the  PP's  are  used  for  well- 
defined  operations  such  that  PP  programs  are  written,  assembled,  and 
tested  well  before  they  are  eligible  for  use  by  a  CP  program.    PP  programs 
extend  the  hardware,  providing  a  virtual  machine  environment  for  user 
processes  that  execute  on  a  CP.    (PP  programs  can  only  be  loaded  from  the 
system  library  of  PP  programs;  thus,  a  user  cannot  redefine    his  own  PP 
routines  to  perform  I/O,  etc„) 

The  architecture  lends  itself  well  to  internally  distributed  monitoring. 
A  special  PP  program  to  gather  data,  filter,  combine,  analyze,  and  record 
monitor  observations  can  be  written  and  added  to  the  system  library;  thus, 
a  PP  simulates  a  hardware  monitor  with  probes  into  the  machine  state  tables 
in  the  central  memory.      For  this  architecture,  the  remainder  of  the  host 
software  and  hardware  need  not  be  altered,  since  a  PP  loaded  with  the 
specially-written  monitor  program  needs  no  other  facility  to  monitor  the 
host. 

Internally  distributed  monitoring  techniques  can  be  extremely  cost-effec- 
tive in  many  circumstances.    The  cost  in  additional  hardware  is  non-exis- 
tent in  these  machines,  since  parts  of  a  distributed  system  are  used  to 
implement  the  monitor  itself.    In  some  studies  that  are  classified  as 
internally  distributed  monitors,  special  purpose  hardware  has  been  per- 
manently added  to  a  production  machine,  see  references  20  and  34.    As  a 
performance  monitor,  the  approach  is  ideal  except  in  the  case  of  a  system 
running  under  processor  saturation  (hence,  a  needed  resource  is  removed 
from  the  system  in  order  to  observe  that  system).    Performance  data  on 
program  counter  distributions  can  be  easily  obtained  in  the  Control  Data 
environment  (but  is  much  more  difficult  in  the  PDP  10  or  Univac  1100  envi- 
ronment).   Since  system  tables  are  stored  in  a  central  memory  where  all 
processors  may  inspect  them,  then  machine  state  is  easy  to  record  and 
analyze;  this  allows  the  pseudo  hardware  monitor  to  identify  causal  rela- 
tionships. 
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As  a  diagnostic  testing  device,  the  approach  also  has  considerable  merit. 
The  pseudo  hardware  monitor  can  be  used  to  run  diagnostic  programs  and 
observe  the  results. 

If  the  application  of  a  monitor  is  either  performance  monitoring  or 
diagnostic  testing,  internally  distributed  monitoring  techniques  lend 
themselves  well  to  remote  monitoring.    Control  Data  currently  employs 
this  technique  for  "Remote  Technical  Assistance"  for  customer  engineers 
at  the  site  of  the  host.       The  monitor  is  stimulated  by  interactive  ter- 
minals through  the  conventional  time  sharing  facilities  of  the  host 
machine. 

As  a  performance  assurance  or  system  security  remote  monitor,  the  appli- 
cation of  internally  distributed  monitors  is  less  attractive,  since  the 
overall  operation  of  the  device  is  ultimately  controlled  by  softwareo 
Again,  if  one  could  ensure  that  complex  software  systems  can  be  proven 
correct  and  that  they  cannot  be  bypassed  or  violated,  then  one  might  be 
less  skeptical  of  these  applications  of  internally  distributed  monitors. 
The  current  state  of  software  engineering  cannot  guarantee  that  either  of 
these  conditions  are  satisfied. 


Programmable  Hardware  Monitors 

Programmable  hardware  monitors  were  introduced  in  the  background  section 
of  this  report;  they  are  characterized  as  hardware  monitors  in  which  the 
operation  of  the  monitor  is  programmable  and,  thus,  dynamically  reconfig- 
urable  as  a  function  of    the  monitoring  environment.    In  order  for  a 
programmable  hardware  monitor  to  be  classified  as  a  remote  monitor,  then 
either  the  programmable  portion  of  the  monitor  must  be  located  at  the 
central  site,  or  else  the  programmable  monitor  must  be  able  to  be  control- 
led from  the  central  site.    This  form  of  hardware  monitor  is,  by  far,  the 
most  popular  type  at  the  current  time  (e.g.  see  references  27,  60,  and  61). 
The  separation     of  hybrid  monitors  from  programmable  hardware  monitors 
is  also  somewhat  arbitrary. 

Sperry  Univac  has  been  heavily  involved  in  the  use  of  programmable  hard- 
ware monitors  for  at  least  ten  years.    In  1969,  a  paper  was  published 
describing  an  1108  monitoring  system  in  which  a  major  component  of  the 
monitor  was  a  second  1108  system;38  currently,  the  Univac  Eagan  Benchmark 
Facility  uses  a  comprehensive  online  monitoring  system  which  heavily 
relies  on  a  Univac  1616  minicomputer,^'^^    The  earlier  system  was  composed 
of  three  components:  a  hardware  monitor  to  gather  data  from  the  host,  data 
collection  software  used  by  the  controller  system  to  record  data  from  the 
hardware  monitor,  and  data  reduction  software  to  analyze  the  collected 
data.    The  initial  goal  for  this  system  was  modest,  namely  to  provide  a 
profile  of  the  program  counter  contents;  however,  the  tools  that  were 
developed  are  clearly  of  a  much  wider  range  of  applicability  than  men- 
tioned in  the  paper.    The  hardware  monitor  component  (i.e.  the  remote 
monitor)  was  simple  circuitry  to  detect  unconditional  branch  instruction 
executions  and  to  record  the  program  counter  contents.    A  critical  obser- 
vation is  that  the  remote  monitor  ran  at  the  same  clock  cycle  as  the  host 
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CPU;  and,  in  fact,  was  driven  off  of  the  same  power  supply  and  clock  as  the 
host  system.    (The  remote  monitor  was  mounted  on  a  single  card  rack  inside 
the  host.) 

Data  collection  was  accomplished  by  a  distinct  1108  system,  with  remote 
monitor  inputs  arriving  via  a  channel,  subsequently  being  recorded  on 
drums  private  to  the  controlling  system.    Monitor  data  were  subsequently 
moved  onto  magnetic  tapes  before  analysis. 

At  first  impression,  the  use  of  an  1108  as  a  central  site  controller 
seems  to  be  a  case  of  overkill.    However,  one  should  note  that  the  combi- 
nation of  central  and  remote  monitor  is  potentially  operating  at  the 
cycle  time  of  the  host;  and,  hence,  the  1108  controller  may  be  required 
in  order  to  record  all  data  that  were  measured. 

The  Univac  Eagan  Benchmark  Facility^'^^  monitor  is  composed  of  a  data 
collection  module,  a  BMD-1100  system,  online  displays,  and  an  operator 
interface.    A  similar  facility  has  been  built  for  the  Univac  London 
Benchmark  Facility.*    The  purpose  of  these  facilities  is  to  provide  online 
monitoring  displays  of  a  wide  variety  so  that  potential  customers  can  see 
the  effect  of  their  workload  on  the  benchmark  machine  (i.e.  an  1100/22 
system) . 

The  data  collection  module  is  made  up  of  twenty  24-bit  comparators,  196 
count/time  registers  with  probe  plugboard  logic  to  support  up  to  400 
standard  Comten  high  impedance  probes.    Thus,  this  portion  of  the  monitor 
corresponds  to  a  conventional  hardware  monitor's  combination  and  filter 
unit.    Monitor  data  can  be  transferred  into  the  BMD-llOO  system  via  an 
I/O  channel  about  once  every  second. 

The  BMD-llOO  (Telecon)  component  is  capable  of  building  real  time  color 
displays,  "replaying"  sketches  of  a  benchmark  run  with  different  displays, 
interacting  with  the  user,  etc.    It  incorporates  a  Univac  1616  minicompu- 
ter (750  ns,  16-bit  memory),  a  tape  drive  and  a  cartridge  disk.  The 
present  uses  of  the  1616  minicomputer  requires  only  about  10  percent  of 
the  CPU  cycles  to  accomplish  data  collection  and  display. 

The  online  displays  at  the  Eagan  facility  include  a  large  color  CRT  dis- 
play with  a  wide  variety  of  display  options,  a  digital  display  of  certain 
critical  elements  of  the  host  system's  status,  and  a  U-100  CRT  terminal 
to  provide  an  operator's  interface. 

The  current  implementations  at  Eagan  and  London  use  the  facility  as  a 
local  hardware  monitor;  however,  either  facility  is  well -suited  as  a 
remote  monitoring  system  with  no  hardware  modification.    Inasmuch  as  the 


*  The  BMD-llOO  system  is  replaced  by  a  Univac  Telecon  system.    The  Telecon 
system  is  a  standard  front  end  processor  for  telecommunication  applica- 
tions, where  the  principal  processing  is  carried  out  by  a  Univac  1616 
minicomputer;  it  is  similar  to  the  special -purpose  BMD-llOO  system. 
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processing  element  is  actually  an  interactive  front  end  processor,  it  is 
obvious  that  the  instrumentation  hardware  and  Telecon  front  end  can  be 
employed  as  a  remote  monitor,  while  the  displays  and  operator's  interface 
are  implemented  at  the  central  site.    Notice  that  the  Telecon  processor 
already  includes  appropriate  hardware  and  software  for  interacting  be- 
tween the  operator's  interface  and  the  remote  processor.    Although  such  an 
installation  relies  on  a  partial  software  solution  (at  the  remote  site), 
a  secure  implementation  might  employ  ROM  for  storing  Telecon  programs,  with 
RAM  used  as  buffer  space.    The  data  collection  and  forwarding  codes  ought 
to  be  simple  enough  to  be  made  convincingly  correct. 

Texas  Instruments  employed  an  in-house  programmable  hardware  monitor  sys- 
tem during  the  development  of  their  ASC  system. ^3, 44, 45, 62, 63    jhe  reason 
for  building  this  monitor  was  performance  evaluation.    The  Texas  Instru- 
ments System  Activity  Monitor  (TISAM)  design  was  motivated  in  part  by  the 
need  for  a  flexible  hardware  monitor  which  could  provide  a  good  monitor- 
analyst  interface.    The  main  component  of  the  system  is  a  TI  960A  mini- 
computer system  with  a  card  reader,  magnetic  tape,  and  interactive  (hard 
copy)  terminal.    Although  the  system  is  capable  of  performing  a  wide  var- 
iety of  measurement  tasks,  it  has  been  used  primarily  for  program  counter 
tracing. 

There  are  a  number  of  other  programmable  hardware  monitors  reported  on  in 
the  open  literature,  e.g.  see  reference  3.    However,  the  above  discussion 
adequately  discusses  the  approach  that  one  might  use  in  applying  the 
technique  to  remote  monitoring. 

Programmable  hardware  monitors  appear  to  offer  significant  promise  as 
remote  monitors  for  most  application  areas  considered  in  this  report. 
As  remote  performance  monitors,  the  approach  has  already  been  successfully 
used  for  as  long  as  ten  years.    The  intelligent  console  applications 
(discussed  later  in  the  report)  are    a  variant  of  programmable  hardware 
monitors,  and  also  have  been  successfully  employed  as  remote  diagnostic 
testing  toolSo    As  performance  assurance  monitors,  the  picture  is  not 
quite  so  clear;  this  area  has  not  yet  been  successfully  accomplished  (to 
this  author's  knowledge).    However,  for  performance  assurance  applications, 
the  use  of  the  programmable  hardware  monitor  at  either  the  local  or  remote 
site  seems  to  have  a  better  chance  at  remaining  secure  than  either  of  the 
previous  two  approaches.    Since  the  remote  portion  of  the  monitor  can  be 
simple,  its  programs  can  be  simple,  and  physically  protected  by  storing 
the  code  in  ROM  and/or  physically  sealing  the  monitor.    In  this  latter 
case,  RAM  memory  might  be  loaded  only  with  some  special  apparatus,  which 
itself  may  be  geographically  distinct  from  the  remote  monitor.    As  a 
system  security  monitor,  it  is  apparent  that  this  general  approach  is 
being  used  in  the  WASSO  terminal  described  in  a  previous  subsection. 


Hybrid  Monitors 

The  di stinction  between  hybrid  monitors  and  programmable  hardware  monitors 
is  principally  one  of  application;  the  components  for  many  programmable 
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hardware  monitors  are  sufficient  to  perform  hybrid  monitorinq.  A  hybrid 
monitor  is  composed  of  a  software  portion  that  executes  on  the  host  hard- 
ware, and  a  hardware  portion  that  executes  a  (usually  programmable)  hard- 
ware monitor.  The  two  portions  are  cognizant  of  one  another  and  exchange 
signals  and  data.  Thus,  a  hybrid  monitor  reacts  to  the  state  of  the  host 
system  by  reconfiguring  itself  dynamically.  It  is  easy  to  argue  that  the 
proposed  WASSO  terminal  and  the  BMD-1100  systems,  as  well  as  others,  are 
really  hybrid  monitors. 

One  early  application  of  the  hybrid  monitor  is  a  remote  tool  was  given 
by    Grochow  in  his  graduate  work  at  MIT.^'    The  Graphic  Display  Monitor- 
ing System  (GDM)  was  designed  to  observe  activity  in  the  GE  645-  Multics 
system  developed  at  Project  MAC.    The  monitoring  functions  were  distri- 
buted across  a  remotely  controlled  software  monitor  and  a  central  site 
programmable  facility.    The  host  software  portion  of  the  monitor  executed 
as  a  multics  procedure,  gathering  data  from  static  operating  system  tables. 
This  procedure  was  also  capable  of  preliminary  filtering  of  the  data  and 
transmission  of  the  partially  filtered  data  to  the  central  site  program- 
mable device.    The  software  portion  was  driven  by  commands  from  the  cen- 
tral site  machine  over  one  data  path,  and  a  separate  data  path  was  used 
for  monitor  data  transmission.    The  host  machine  central  site  facility 
interface  was  implemented  via  two  data  channels  on  the  GE  645  to  a 
2400  baud  modem,  through  a  telecommunications  link  to  a  similar  modem  at 
the  central  site.    The  central  site  mechanism  was  composed  of  a  DEC  PDP  8 
system  including  a  disk,  magnetic  tape  and  a  display  processor.  The 
central  site  portion  of  the  system  used  much  of  its  computing  capability 
to  format  real  time  displays,  such  as  the  usage  of  Multics  core  memory 
pages,  the  multi -programming  state  of  each  process  in  the  system,  etc. 
These  analysis  and  display  programs  existed  in  the  basic  library  of 
functions  of  GDM.    However,  the  GDM  was  also  user    programmable  so  that 
monitor  functions  and  displays  could  be  generated  whenever  new,  or  unique, 
requirements  were  encountered. 

The  GDM  was  found  to  be  useful  in  its  flexibility  and  extensibility.  The 
software  portion  of  the  monitor  was  constructed  as  a  sampling  monitor, 
with  the  sampling  rate  determined  by  requests  from  the  PDP  8  (thus,  the 
sample  rate  was  taken  as  a  function  of  the  display  being  used).  The 
2400  baud  transmission  rate  for  host-PDP  8  interconnection  allowed  for  a 
maximum  of  about  20  samples  per  second,  although  many  displays  were 
changed  only  once  a  second. 

In  1971,  Aschenbrenner,  Amiot,  and  Natarajan  published  a  paper  briefly 
describing  their  Neurotron  monitor  system  (implemented  at  Argonne  National 
Laboratory).^    The  monitor  incorporates  a  minicomputer  to  control  the 
combination  and  filer  unit,  as  well  as  data  filtering  and  recording. 
The  external  monitor  and  the  host  software  monitor  interact  via  conven- 
tional I/O  ports,  as  well  as  through  measurement  probes.    The  development 
goals  of  Neurotron  offer  several  interesting  facets  to  the  use  of  the 
system  for  remote  monitoring.     Among  other  goals,  the  monitor  was 
designed  to: 

-    contain  program  logic  for  selecting  or  filtering  events  as  a 
function  of  the  current  experiment 
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-  allow  micro  and  macro  level  monitoring  of  the  host 

-  trigger  monitoring  activity  as  a  function  of  host  event 
sequences  or  at  predefined  time  intervals. 

It  is  not  clear  exactly  how  successful  the  Neurotron  monitor  was  after  it 
had  been  used  for  an  extended  period  of  time.    The  Neurotron  was  used  to 
gather  statistics  on  instruction  distribution  and  correlate  them  with 
I/O  activity  or  other  events,  to  investigate  memory  reference  streams, 
to  analyze  buffer  usage,  etc.    Perhaps  the  most  interesting  claim  made 
about  the  Neurotron  is  the  portability  of  the  hardware  to  different 
systems,  each  containing  a  host  software  monitor.    Apparently,  a  single 
hardware  monitor  can  be  constructed  that  will  successfully  operate  with  a 
variety  of  host  systems. 

In  the  late  1960's  and  early  1970's  Xerox  Data  Systems  was  deeply  involved 
in  measurement  projects  for  operating  systems  and  the  Sigma  series  hard- 
ware.   Hughes  and  Cronshaw  describe  another  hybrid  monitor  that  was  used 
within  XDS.21    The  ADAM  hybrid  monitor  differs  little  from  the  Neurotron 
monitor  in  its  organization;  the  facility  incorporated  a  minicomputer  in 
the  hardware  monitor.    Interactions  between  the  host  and  the  hardware 
monitor  took  place  over  an  I/O  channel,  while  the  hardware  monitor  could 
also  collect  data  from  probes  installed  on  the  host.    The  most  signifi- 
cant differences  between  the  ADAM  and  the  Neurotron  were  in  the  event 
recognition  algorithms  and  their  versatility.    ADAM  used  an  associative 
memory  to  quickly  recognize  event  combinations;  secondly,  the  Xerox 
system  was  intended  only  to  monitor  the  Sigma  7,  while  the  Neurotron  was 
aimed  at  a  variety  of  different  host  systems. 

The  technology  of  hybrid  monitors  corresponds  closely  to  that  of  program- 
mable hardware  monitors.    The  strong  points  of  each  are  similar;  e.g. 
careful  design  of  the  remote  portion  of  a  hybrid  or  programmable  hardware 
monitor  is  likely  to  be  more  "trustworthy"  than  for  pure  software  tech- 
niques that  execute  solely  on  the  host  hardware. 

Network  Monitors 

Network  monitors  are  defined  to  be  any  monitoring  device  used  in  the 
context  of  a  distributed  computer  network.    The  primary  motivation  for 
monitoring  in  this  environment  has  been  for  performance  evaluation.  By 
the  nature  of  computer  networks,  nearly  all  measurement  studies  must 
employ  a  remote  monitor. 

Perhaps  the  most  prominent  computer  network  in  the  world  is  the  ARPANET 
which  has  been  operational  since  about  1971o    The  ARPANET  contains  in 
excess  of  50  significant  (i.e.  medium  to  large  scale)  computer  nodes,  most 
intercommunicating  at  a  rate  of  50  KBPS.    There  are  two  network  centers 
devoted  to  measurements:  the  Network  Control  Center  at  Bolt,  Beranek,  and 
Newman,  and  the  Network  Measurement  Center  at  UCLA.    A  few  papers  have 
appeared  in  the  open  literature  that  describe  measurement  activity  on 
the  ARPANET.    Kleinrock  and  Naylor  provide  a  report  on  investigations 
dealing  with  message  traffic  in  the  network. Each  node  in  the  ARPANET 
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contains  an  interface  processor  called  an  IMP.    The  IMP  is  a  fully  program- 
mable 16-bit  minicomputer  whose  primary  function  is  to  interface  the  host 
processor  to  the  ARPANET  in  a  standard  manner.    The  IMP  must  also  parti- 
cipate in  message  packet  switching.    Each  IMP  contains  software  to  aid 
in  network  measurements;  for  example  the  IMP  can  collect  data  about 
its  host  and  forward  the  monitor  data  to  the  Network  Measurement  Center. 
The  IMP  monitor  software  also  may  "time  stamp"  packets  which  they  forward 
to  other  IMP's,  thus  allowing  an  analysis  program  to  investigate  message 
packet  flow  rates  and  routes.    The  standard  ARPANET  remote  monitor  is 
really  a  software  monitor  implemented  on  an  IMP,  but  under  the  control  of 
the  Network  Control  Center.    Another  view  of  the  facility  is  that  it  is 
functionally  equivalent  to  the  Univac  Telecon  monitoring  facility  discus- 
sed earlier;  i,e.  the  IMP  corresponds  to  a  Univac  Telecon  front  end  pro- 
cessor (Univac  1616  minicomputer). 

Another  monitor  has  also  been  developed  for  investigating  an  alternate 
packet  switching  mechanism  ("packet  radio  systems").^'    In  this  discus- 
sion a  packet  radio  repeater,  rather  than  an  IMP,  is  used  to  broadcast 
packets  with  a  radio  transceiver.    The  packet  radio  repeater  also  inclu- 
des a  microprocessor  to  aid  in  the  broadcast  and  to  act  as  a  monitoring 
facility.    In  order  to  avoid  the  halo  effect  of  broadcasting  the  measure- 
ment data  over  the  normal  data  channel,  monitor  data  is  saved  at  the 
station  until  the  measurement  experiment  is  completed.    It  may  then  be 
transmitted  to  a  central  facility  for  further  processing.    Another  inter- 
esting aspect  of  the  work  is  that  the  packet  radio  repeater  microproces- 
sor does  not  maintain  a  local  library  of  monitoring  routines;  instead, 
whenever  a  measurement  test  is  to  be  initiated,  the  monitoring  code  is 
transmitted  to  the  packet  radio  repeater„    In  the  case  of  performance 
assurance  and  system  security  monitors,  this  idea  has  obvious  interesting 
possibil ities. 

A  final  note  on  ARPANET  remote  monitoring  is  concerned  with  measurements 
of  the  London  node  of  the  network. 50    a  PDP  9  is  used  to  collect  measure- 
ment data  on  the  IBM  360/195  and  then  analyzed  as  batch  data  after  the 
data  collection  phase. 

Considerable  thought  has  gone  into  the  topic  of  remote  monitors  for  net- 
works at  the  University  of  Waterloo.-^'    At  the  time  that  the  paper  was 
published,  the  group  had  built  a  prototype  Computer  Network  Monitoring 
System  (CNMS)  designed  to: 

-  observe  network  performance 

-  detect  malfunctions  in  the  network 

-  diagnose  failures 

A  basic  premise  of  the  measurement  group  was  that  software  monitors, 
alone,  often  produced  too  much  artifact  (cf.  ARPANET  studies),  and  that 
pure  hardware  monitors  were  not  sufficiently  flexible  to  make  the  required 
network  measurements.    Therefore,  a  hybrid  monitor  design  would  be  used. 
The  general  approach  dictated  that  each  host  computer  should  include  a 
remote  monitor  to  be  software  controlled  from  the  network  monitoring 
center. 
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The  important  components  of  the  CNMS  are  a  remote  controlled  hybrid  moni- 
tor (RCHM)  and  the  central  site  network  measurement  center  (NMC  as  in 
the  ARPANET).    RCHM's  are  potentially  allowed  to  communicate  with  the  NMC 
either  through  the  normal  network  data  paths  or  through  dedicated  links. 
The  philosophy  also  admits  to  a  hierarchy  of  measurement  centers;  e.g. 
each  RCHM  could  be  controlled  by  a  regional  NMC  (RNMC)  which  is,  in  turn, 
controlled  by  the  NMC. 

The  prototype  RCHM  is  based  on  a  PDP  11  (probably  an  LSI-11)  processor 
with  extra  hardware  to  implement  a  combination  and  filter  unit  and  an 
event  recognition  unit.    RCHM  is  attached  to  the  host  computer  with 
probes  and  through  the  conventional  I/O  communication  lines  of  the  host. 
The  PDP  11  system  includes  a  small  disk  for  buffering  raw  data,  which  can 
be  forwarded  to  a  (R)NMC  in  a  condensed  form. 

The  software  portion  of  an  RCHM  is  distributed  across  the  PDP  11  CPU  and 
the  CPU  of  the  host  computer.    Host  resident  monitor  code  is  system- 
dependent  with  system-independent  interfaces  (between  the  RCHM-resident 
software  and  the  host-resident  software).    The  RCHM-resident  software  is 
organized  as  a  set  of  six  classes  of  processes,  functions  of  the  classes 
being: 

-  experiment  manager  to  schedule  and  support  RCHM  experiment 
control  programs 

-  monitor  manager  to  control  the  special -purpose  monitoring 
hardware  (probes,  filters,  etc.) 

-  resource  manager  to  allocate  RCHM  system  resources 

-  communications  manager  to  receive  input  from,  and  forward 
monitor  data  to,  its  (R)NMC 

-  results  manager  procedures  to  record,  reduce,  and  analyze  raw 
monitor  data,  producing  condensed  reports 

-  maintenance  manager  to  provide  diagnostic  testing  codes 
for  the  network  components  (i.e.  the  host,  the  RCHM,  etc.) 

The  (R)NMC  system  is  not  a  special-purpose  computer  system  (although  it 
must  have  a  hardware  facility  to  communicate  with  the  set  of  RCHM's  that 
it  controls).    The  (R)NMC  software  contains  seven  classes  of  processes 
that  roughly  complement  the  six  classes  of  RCHM  processes;  the  seventh 
class  establishes  a  user  interface  with  the  analyst  who  controls  the 
experiment. 

An  instance  of  a  measurement  experiment  in  CNMS  might  proceed  as  follows: 
The  purpose  of  the  experiment  is  carefully  analyzed,  and  a  set  of  meas- 
urements that  will  provide  the  necessary  results  is  determined.  The 
RCHM's  are  then  configured  to  take  the  required  measurements  by  attach- 
ing hardware  probes,  designing  and  implementing  host  software  probes, 
and  designing  and  implementing  RCHM-resident  data  collection  and  reduc- 
tion software.    Since  each  RCHM  hardware  system  is  identical,  one  set  of 
software  is  written  for  all  remote  monitors.    The  codes  should  be  able 
to  collect  bursts  of  measurement  data  from  the  hardware  and  software 
probes,  record  the  data,  condense  the  data  (usually  to  a  histogram)  and 
then  transmit  the  condensed  monitor  data  to  the  controlling  (R)NMC. 
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Once  the  monitors  have  been  defined,  the  experiment  can  be  initiated  from 
the  NMCo    Online  control  can  be  exercised  by  the  analyst  via  the  user 
interface  manager  that  executes  on  the  NMC.    The  interaction  is  possible 
because  of  real-time  analysis  and  display  at  the  NMC.    After  an  experiment 
is  initiated  in  the  RCHM's,  there  is  no  requirement  that  the  analyst  con- 
tinue to  monitor  the  experiment  from  the  NMC;  the  communication  between 
RCHM's  and  the  {R)NMC  need  not  be  continuous.    Upon  conditions  determined 
by  the  NMC  or  the  analyst,  the  experiment  can  be  terminated. 

Xerox  Palo  Alto  Research  Center  has  also  developed  monitoring  tools  for 
investigating  their  internal  network.^"   The  network  connects  a  number 
of  minicomputers  and  at  least  one  medium-scale  computer  system.    Each  of 
the  minicomputer  systems  can  be  loaded  with  a  remotely  controlled  soft- 
ware monitor.    The  monitoring  facility  provides  standard  software  for 
recording  data  and  for  transmitting  it  to  other  nodes  on  the  network.  The 
work  is  similar  to  the  facilities  provided  in  the  ARPANET,  and  not  nearly 
as  wel 1 -developed  as  those  described  in  the  CNMS. 

Tesdata  Systems  Corporation  has  two  particular  monitor  systems  that  can 
be  classified  as  network  monitors.    The  MSB  facility  is  a  product,  aimed 
at  smal 1 -to-medium  sized  data  centers,  for  sharing  more  sophisticated  and 
expensive  monitoring  equipment  with  other  centers  of  similar  size.^^ 
The  approach  taken  in  the  MSB  is  to  install  a  programmable  hardware  mon- 
itor at  each  remote  site.    The  hardware  monitor  is  based  on  a  16-bit  micro- 
processor with  RAM  for  buffer  storage  and  a  fixed  set  of  (not  more  than 
64)  standard  hardware  probes.    At  intervals  of  time  determined  by  the 
measurement  sampling  rate,  a  central  site  controller  initiates  communica- 
tion with  the  remote  monitor  to  dump  the  remote  buffers.    Data  collected 
by  the  central  controller  may  or  may  not  have  been  filtered  by  the  remote 
microprocessor.    The  remote  monitor  is  designed  with  ROM  and  RAM  memory. 
The  ROM  contains  several  built-in  functions  for  the  microprocessor,  while 
the  RAM  is  used  for  buffering  and  dynamic  portions  of  the  monitor  code. 
The  RAM  portion  of  the  microprocessor  is  down  loaded  from  the  central 
site  controlling  processor.    Central  site  computing,  performed  at  a 
regional  control  center,  produces  summary  reports  on  a  conventional 
computer  system. 

The  distributed  measurement  network  is  a  production  facility  of  Tes- 
data.      The  network  consists  of  several  secondary  systems  made  up  of 
conventional  Tesdata  monitors,  e.g.  MS  38  or  MS  58  monitor  systems,  and 
a  single  primary  system  implemented  as  an  MS  88  monitor  system.  The 
secondary  systems  are  connected  to  the  primary  system  via  standard  voice 
grade  phone  lines,  and  are  controlled  by  the  MS  88  system.    The  differen- 
ces between  the  distributed  measurement  network  and  the  MSB  are  quite 
significant.    The  secondary  systems  can  operate  as  autonomous  monitors  or 
be  a  node  in  a  monitoring  network.    Each  secondary  system  contains  com- 
plete data  collection  and  archiving  equipment,  analysis  programs,  and  re- 
porting mechanisms.    An  MSB  remote  monitor  is  not  capable  of  operating 
totally  independently.    The  distributed  measurement  network  is  aimed  at 
collections  of  large  data  centers. 


National  Bureau  of  Standards  has  its  own  Network  Measurement  Machine  (NMM) 
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and  Network  Measurement  System  (NMS)„    The  PDP  11  based  NMM  is  a  hybrid 
remote  monitor  controlled  by  central  site  facilities  of  the  NMS.  Consult 
references  1,  2,  and  39  for  further  information  on  this  worko 

Finally,  Lombardi  has  proposed  a  network  monitoring  facility  similar  to 
the  MSB  system  described  above,  although  implementation  had  not  been 
completed  in  September,  1977,^° 

Network  monitors  are  the  archetype  for  remote  monitors;  the  nature  of 
networks  implies  that  remotely  located  computer  systems  and  network 
traffic  can  best  be  monitored  by  the  combination  of  a  remote  monitor  and 
a  central  control  element.    Remote  monitoring  of  a  single  computer  system 
is  merely  a  degenerate  case  of  network  monitoring.    As  remote  monitors 
for  performance  evaluation,  network  monitors  are  already  in  production, 
e.g.  Tesdata  facilitieSo    The  CNMS  system  additionally  uses  network 
monitoring  techniques  for  providing  a  diagnostic  testing  facility.  Appli- 
cations in  the  areas  of  performance  assurance  and  system  security  are  not 
tested  (even  for  local  systems,  performance  assurance  and  system  security 
have  not  been  successfully  implemented) „    Of  all  the  remote  monitoring 
classifications  discussed  up  to  this  point,  it  is  easiest  to  argue  for  the 
probability  of  a  successful  implementation  using  these  techniques.  In 
performance  assurance  and  system  security  applications,  critical  questions 
of  the  monitoring  activity  are  the  independence  of  the  remote  monitor  from 
the  host  and  the  validity  of  the  remote  monitor.    The  possibility  of  build 
ing  logically  compact  hybrid  remote  monitors  which  can  be  proven  correct 
offers  some  hope  for  the  latter  question;  the  use  of  ROM  for  storing 
remote  software  also  aids  in  ensuring  validity  of  the  monitor..  Distribu- 
ting the  remote  monitor  across  hardware  and  software  aids  in  creating  inde^ 
pendence  between  the  host  and  the  remote  monitor.    These  factors  are  an 
obvious  point  for  further  research. 

Fault  Diagnosis  Monitors 

Much  work  has  been  done  in  the  area  of  fault-tolerant  computing,  espe- 
cially in  the  area  of  circuit  verification.    A  fault-tolerant  computer 
system  is  one  in  which  a  single  error  will  not  cause  the  system  to  fail; 
the  overall  system  can  detect  and  correct  the  error.    Some  obvious  exam- 
ples of  fault  detection  monitoring  occur  in  fault-tolerant  computers  such 
as  in-flight  or  on-board  computer  systems. 

Unfortunatly,  most  work  in  the  area  of  fault  diagnosis  is  at  a  rather 
detailed  level.    Hardware  faults  usually  occur  at  the  gate  level;  hence, 
monitoring  equipment  must  gather  data  at  that  level „    Thus,  fault  diagno- 
sis monitors  tend  to  gather  such  detailed  information  that  the  data  is 
only  useful  for  circuit  analysis  or  redundant  processing  (e.g.  see  refer- 
ence 10)  and  is  not  generally  useful  for  system  level  monitoringo 

One  special-purpose  diagnostic  facility  that  has  been  employed  in  indus- 
try is  the  Control  Data  STAR  Maintenance  Station.^'    Although  the  mainten-[ 
ance  station  appears  to  be  at  least  as  applicable  of  a  diagnosis  tool  as 
the  PP  in  the  6000  series  machines.  Control  Data  personel  indicate  that  it 
has  not  been  a  useful  addition  to  the  STAR  facil ities. ^ ^ 

20 


Related  work  takes  place  in  other  areas  of  hardware-diagnosis  using 
extended  system  consoles,  and  that  work  will  be  discussed  in  the  next 
subsection. 


Intelligent  and  Extended  Consoles 

The  final  category  of  remote  monitors  uses  the  notion  of  the  intelligent 
terminal  for  an  operator's  console,.    It  is  frequently  useful  to  provide 
more  capability  in  the  operator's  console  than  exists  in  a  passive 
terminal;  e.g.,  the  console  can  be  used  to  drive  CRT  displays  in  parallel 
with  the  operation  of  all  other  facilities  in  the  host  system. 

The  oldest  application  of  the  technique  must  be  that  of  the  Control 
Data  6600  consoleo^'^    One  of  the  ten  peripheral  processors  is  dedicated 
to  the  task  of  driving  the  dual  CRT  system  console.    The  PP  reads  input 
from  the  operator's  keyboard  and  from  the  machine's  central  memory,  and 
processes  the  data  to  produce  CRT  displays  describing  the  state  of  the 
operating  system. 

The  Honeywell  Remote  Maintenance  System/62  (RMS-62)  is  a  small  system 
tool  to  aid  field  engineers  in  monitoring,  controlling,  diagnosing,  and 
patching  Level  62  installations. ^^'^^    The  remote  portion  of  the  device 
is  a  Remote  Console  Interface  Adaptor  and  a  software  diagnostic  package. 
The  adaptor  is  a  single  board  component  which  provides  a  300  baud  modem 
to  send/receive  information  to/from  the  host.    The  modem  is  attached  to 
the  bus  connecting  the  host  system  and  its  console.    When  the  host  is 
experiencing  problems,  the  adaptor  is  connected  to  a  central  site  system 
console  via  voice  grade  phone  lines.    Diagnostic  software  is  then  exer- 
cised from  the  central  site. 

Digital  Equipment  Corporation  has  announced  a  facility,  similar  to  the 
RMS-62,  for  use  with  their  PDPll/70.47    Few  details  of  the  work  are  avail- 
able since  the  project  is  still  under  proprietory  development.  However, 
the  adaptor  contains,  not  only  a  modem  for  communicating  with  the  central 
controller,  but  also  a  microprocessor  with  a  small  amount  of  RAM,  The 
unit  is  to  be  used  only  for  diagnostic  testing.    One  difference  between 
the  RMS-62  and  the  DEC  system  is  that  the  DEC  facility  allows  communication 
with  either  the  host's  local  operator  console  or  the  central  site;  in  the 
latter  case,  the  microprocessor  is  required  to  do  some  console  processing 
at  the  site  of  the  host.    Digital  Equipment  personnel  speculate  that  the 
facility  is  too  slow  to  be  used  as  a  performance  monitor  and  will  only  be 
used  as  a  diagnostic  facility. 

The  Cray  system  also  employs  an  intelligent  operator's  console  in  its 
standard  configuration.^^    The  console  is  driven  by  a  Data  General  Eclipse 
minicomputer  (a  400  nanosecond,  16-bit  minicomputer)-    The  console  is 
also  used  as  an  initial  program  loading  device  and  for  diagnostic  testing. 
Whereas  the  DEC  system  extended  console  saturates  its  processing  facility, 
the  Cray  system  tends  to  under  utilize  its  console  processor. 
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The  Amdahl  470  computer  system  also  uses  a  Data  General  minicomputer  as 
a  processor  to  drive  its  console,^    The  console  hardware  includes  a 
floppy  disk  unit  and  a  modem  for  telecommunication,  as  well  as  facilities 
to  support  the  operator's  console. 

The  circuit  components  that  are  used  to  build  the  470  include  the 
capability  for  an  external  medium  to  read  out  the  value  of  each  latch  in 
the  hardware.    The  first  extended  capability  of  the  console  is  to  be  able 
to  inspect  and  record  each  of  these  latches.    These  console  facilities 
are  used  whenever  the  hardware  fails,  i.e.  the  console  scans  all  latches 
in  the  machine  and  records  them  on  a  floppy  disk.    The  floppy  disk  is  then 
mailed  to  the  Sunnyvale  site  where  it  can  be  analyzed  by  central  site 
experts . 

The  second  application  of  the  extended  console  uses  the  (1200  baud)  modem. 
If  a  machine  is  experiencing  a  problem  that  the  field  engineer  cannot 
diagnose,  he  initiates  a  phone  connection  of  the  console/host  machine 
to  the  Sunnyvale  (central)  facility.    At  the  central  site,  another  console 
with  a  mini  computer  is  used  to  remotely  control  the  host  computer  system. 
Thus,  there  is  no  real-time  monitoring  at  1200  baud,  but  incremental 
stepping  of  the  host  can  be  done  from  the  central  site.    (Diagnostic  soft- 
ware is  executed  on  the  remote  console  processor  under  the  control  of  the 
similar  central  console  processor.) 

The  third  application  of  the  console  extends  the  one  just  described. 
After  a  connection  is  established  between  the  host  console  and  the  cen- 
tral console,  information  routed  to  the  central  console  is  processed 
locally  by  a  second  470  system.    Full  trace  data  can  be  analyzed  by  the 
central  facility  in  order  to  isolate  circuit  failures. 

The  final  example  of  an  extended  console  is^the  Total  Remote  Assistance 
Center  (TRACE)  facility  used  at  Univac.-^'-''^^  The  central  facility  is 
centered  around  a  Univac  1108  in  Roseville,  Minnesota,  which  is  dedicated 
to  diagnostic  assistance  for  field  engineers.    The  central  facility  can 
be  used  to  remotely  monitor  series  90  systems  and  1100  series  systems. 
(The  remote  monitor  for  each  machine,  in  each  series,  is  unique  to  that 
machine  model . ) 

The  series  90  remote  monitors  are  portable  facilities  used  only  during 
diagnostic  testing.    The  1110  and  1100/40  are  provided  with  a  Maintenance 
Controller  to  serve  as  the  remote  (programmable  hardware)  monitor.  The 
Maintenance  Controller  provides  a  mechanism  for  extending  the  console  to 
the  TRACE  facility  and  to  run  diagnostic  tests.    The  1100/80  remote 
monitor  contains  slightly  more  capability  than  the  Maintenance  Controller, 
allowing  more  of  the  TRACE  processing  to  be  performed  at  the  site  of  the 
host. 

The  TRACE  facility  bears  many  similarities  to  the  network  monitors  dis- 
cussed in  the  previous  subsection.    Although  Control  Data  did  not  feel 
that  dedicated  remote  monitors  were  useful  in  the  context  of  STAR,  Univac 
has  made  extensive  use  of  the  same  technique  (so  much  so  that  future 
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Univac  systems  will  also  include  similar  facilities'*'^).    The  most  remark- 
able observation  about  the  TRACE  facility  is  its  versatility  across  a 
wide  variety  of  machines;  the  remote  monitors  have  been  designed  for 
individual  models  so  that  uniform  central  site  processing  can  be  applied 
to  the  monitor  data. 

The  remarks  at  the  end  of  the  subsection  on  network  monitors  are  appro- 
priate for  most  extended  console  applications  discussed  here.    The  exten- 
ded consoles  are  often  equivalent  to  the  two-node  network  (consisting  of 
the  host  system  and  the  central  system).    As  performance  evaluation  mon- 
itors, the  extended  console  may  have  inadequate  local  processing  power 
or  be  limited  in  its  data  transmission  bandwidth  (e„g.  the  Honeywell  and 
Digital  Equipment  facilities).    Almost  all  extended  consoles  have  been 
specifically  designed  to  perform  remote  diagnostic  capabilities;  thus, 
they  are  nearly  all  successful  at  this  aspect  of  remote  monitoring.  As 
performance  assurance  and  system  security  remote  monitors,  the  question 
of  appropriateness  is  still  open  (as  mentioned  before). 


SUMMARY  AND  CONCLUSIONS 

This  report  has  discussed  a  wide  variety  of  monitors  in  the  context  of 
remote  monitoring.    The  areas  of  application  have  been  broadly  categor- 
ized as  performance  evaluation,  diagnostic  testing,  performance  assurance, 
and  system  security  monitoring.    Monitors  have  been  classified  into  seven 
groups,  based  primarily  on  their  architecture.    Several  individual  monitors 
could  easily  have  been  placed  in  more  than  one  category  (the  categor- 
ization really  served  to  compare  similar  monitors  at  one  time  in  a  single 
subsection) . 

There  is  no  single  best  monitoring  approach  for  any  given  application. 
The  choice  of  a  technique  is  a  function,  not  only  of  the  broad  application 
area,  but  also  of  the  particular  environment  in  which  the  monitor  is  to 
operate  and  the  type  of  data  that  the  monitor  will  be  required  to  collect. 
Nevertheless,  the  following  paragraphs  address  the  issue  In  very  general 
terms. 

Remote  monitoring  for  performance  evaluation  is  the  most  popular  applica- 
tion area.    The  monitoring  technology  has  grown  complex  during  the  ten  or 
fifteen  years  that  local  performance  monitors  have  been  used,  propagating 
a  large  set  of  principles  that  could  be  employed  in  remote  monitoring  work. 
Activity  has  been  the  highest  in  the  study  of  computer  networks,  although 
that  technology  is  clearly  derived  from  software,  programmable  hardware, 
and  hybrid  monitoring  work.    It  is  also  clear  that  current  remote  monitor 
architecture  studies  have  far  outstripped  the  engineering  application  of 
the  attendant  tools.    Existing  hybrid  and  network  monitor  facilities 
appear  to  be  capable  of  gathering  measurement  data  in  manners  that  have 
not  really  been  effectively  used.    The  tool  maker  has  built  a  tool  that 
is  so  complex  that  workers  do  not  know  how  to  use  it  effectively.    A  few 
of  the  toolmakers  seem  to  be  aware  of  this  problem  and  are  attempting  to 
remedy  the  situation  by  providing  better  human-engineered  monitors.  The 
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raw  power  of  the  existing  monitors,  as  performance  monitors,  is  not  likely  P 
to  be  a  limiting  factor  for  some  time  to  come.  S 

The  area  of  diagnostic  testing  using  remote  monitors  is  especially  well  ' 

balanced.    Manufacturers  have  extended  their  console  capability,  or  pro-  j 

vided  other  special  facilities,  to  accomplish  precise  remote  diagnostic  I 
monitors.    The  Amdahl  and  Univac  extended  console  facilities  are  not  only 

wel 1 -designed  (at  least  at  a  high  level),  the  features  of  the  monitors  ) 

are  also  well  utilized.    Although  other  categories  of  monitors  can  be  ( 

used  to  implement  diagnostic  testing,  the  extended  console  appears  to  be  ! 
the  best  approach  to  this  application  area.    Conversely,  extended  consoles 

can  be  extended  to  serve  purposes  other  than  diagnostic  testing,  although  \ 

remote  monitor  (host  console)  processor  saturation  may  be  a  problem.    If  i 

the  host  console  processor  is  embellished  to  include  facilities  for  fil-  J 

tering  and  data  recording,  the  extra  memory  may  be  left  unused  for  large  t 
periods  of  time,  i.e.  the  hardware  investment  may  not  be  cost-effective. 

Performance  assurance  monitoring  is  a  difficult  application  area  in  which 
to  work.    A  fundamental  axiom  of  all  monitoring  studies  is  to  know  what 
data  is  needed  before  designing  a  monitor  to  collect  the  data.    The  ques- 
tion of  whether  or  not  performance  assurance  is  even  possible  is  currently 
unanswered.    If,  indeed,  it  is  a  solvable  problem,  then  one  can  proceed 
with  an  appropriate  design.    For  the  sake  of  discussion,  it  is  assumed 
that  one  may  be  able  to  find  performance  assurance  metrics,  and  that  a 
monitor  is  needed  to  obtain  the  corresponding  measures.    By  the  nature  of 
the  requirement  for  performance  assurance,  one  cannot  assume  that  the 
environment  of  the  host  computer  is  totally  friendly  (for  if  it  were,  per- 
formance assurance  would  be  unnecessary).    The  degree  of  unfriendliness 
can  vary  from  situations  involving,  say,  mischievous  university  students, 
to  the  clientele  of  an  Eastern  European  computing  center.    Hence,  imple- 
mentation issues  must  be  concerned  with  the  correctness,  the  logical 
completeness,  and  the  security  of  the  monitor  (especially  its  remote 
portion). 

Correctness  of  any  design  can  best  be  ensured  by  keeping  the  monitor  ; 
design  as  simple  as  possible  (a  lesson  hard-learned  by  current  operating 
system  designers).    When  the  design  is  simple,  or  at  least  highly  struc-  ] 
tured,  then  its  logical  consistency  can  be  checked  or  proven.  ■ 

Logical  completeness  has  to  do  with  the  absence  of  "loopholes"  in  the 
design  and/or  bypassing  the  monitor  again.    One  would  like  to  have  proof 
of  logical  completeness,  but  this  is  most  likely  impossible.    (For  example,' 
the  IRS  has  been  unable  to  establish  a  set  of  tax  laws  which  effectively 
plug  all  tax  loopholes.)    The  best  approach  toward  achieving  logical  ' 
completeness  is  to,  again,  keep  the  monitor  design  as  simple  and  well-  ; 
structured  as  possible;  only  then  can  a  single  human  being  begin  to 
understand  the  implications  of  the  design  in  terms  of  completeness. 

Issues  of  security  are  at  least  as  difficult  to  address  as  those  of  j 
logical  completeness  (by  a  secure  monitor,  it  is  meant  that  the  monitor  ' 
cannot  be  violated  to  change  its  ability  to  collect  information).  Secure 
remote  monitors  will  almost  necessarily  be  at  least  partially  implemented 
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in  hardware,  since  software  is  extremely  susceptible  to  security  loopholes. 
Hybrid  monitoring  techniques  show  promise  in  terms  of  maintaining  secure 
operation.    To  minimize  the  chance  of  programmable  hardware  being  tam- 
pered with,  the  hardware  monitor  code  should  be  stored  in  ROM  and/or  down 
loaded  from  the  central  facility.    Physical  security  might  be  enhanced 
by  embedding  the  monitor  into  the  other  facilities  of  the  host  system 
rather  than  "locking  it  up"  in  a  box. 

System  security  is  the  main  issue  of  the  WASSO  terminal  prototype 
implementation.    As  in  performance  assurance,  questions  of  monitor 
consistency,  completeness,  and  security  are  critical.    Basically,  the 
same  observations  made  about  performance  assurance  monitors  also  hold  for 
system  security  monitors. 

The  overall  future  of  remote  monitoring  is  filled  with  several  unanswered 
questions.    How  can  remote  performance  monitors  be  used  to  their  maximum 
effectiveness?    Is  performance  assurance  decidable  in  a  practical  sit- 
uation involving  heuristics?    If  the  question  is  decidable,  what  are 
techniques  for  ensuring  some  reasonable  level  of  consistency,  complete- 
nesss,  and  security?    Hardware  solutions  will  probably  tend  to  surface, 
since  such  solutions  are  becoming  less  expensive  and  they  can  be  made  to 
be  less  volatile  than  pure  software  solutions. 
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